Singular Value Decomposition

with σ1 ≥ σ2 ≥ · · · ≥ σr > 0 and r = rank(A). In the above, σ1, . . . , σr are the square roots of the eigenvalues of AA. They are called the singular values of A. Our basic goal is to “solve” the system Ax = b for all matrices A and vectors b. A second goal is to solve the system using a numerically stable algorithm. A third goal is to solve the system in a reasonably efficient manner. For instance, we do not want to compute A using determinants. Three situations arise regarding the basic goal: (a) If A is square and invertible, we want to have the solution x = Ab. (b) If A is underconstrained, we want the entire set of solutions. (c) If A is overconstrained, we could simply give up. But this case arises a lot in practice, so instead we will ask for the least-squares solution. In other words, we want that x̄ which minimizes the error ‖Ax̄ − b‖. Geometrically, Ax̄ is the point in the column space of A closest to b. That is, UU = UU = I .